fix(shader): LDS float atomics quiet signalling NaNs and flush denormals as the hardware does - #2312

What

Follow-up to #573 (the 64-bit LDS atomics), whose TechnicalDebt entry said the f64 forms were not measured on hardware. ds_min/max_f32, ds_cmpst_f32, ds_min/max_f64, ds_cmpst_f64 and their _rtn forms now follow the rule measured on RDNA2 (gfx1036, Ryzen 7950X3D iGPU) with tools/hw-oracle, over 14 × 14 operand sets per type (±0, ±1, ±inf, ±smallest denormal, quiet and signalling NaNs of both signs and with payloads) in IEEE 0 and 1 with every value 0-3 of the type's denormal field (fields 1 and 2 added after review); every memory and returned result of the six f64 and nine f32 results per pair, in each of the eight modes, 0 mismatches:

• both operands are flushed to a signed zero when bit 0 of the type's denormal field is clear (values 0 and 2; 1 and 3 keep them; FLOAT_MODE bits 5:4 for f32, bits 7:6 for f64; a shader without a known mode flushes f32 and keeps f64, as elsewhere);
• a signalling-NaN data operand is stored quieted; else a signalling-NaN old value is stored quieted; else a quiet-NaN old value is replaced by the data; else a quiet-NaN data operand loses; else the ordered compare with −0 below +0 decides; the flushed value is what is written, also when the old value wins;
• cmpst compares the flushed bits (both zeros equal, NaN never equal) and leaves memory unchanged when they differ;
• the returned value is always the old value as it was in memory; IEEE_MODE has no effect.

This is the rule AtomicFloatBits::minMax already implements for the buffer float atomics, plus the flush. The shared f32 path used OpIsNan/OpFOrdLessThan/OpFOrdEqual on the bit-cast values, so its NaN and denormal results were the driver's (both RADV and NVIDIA quiet nothing and flush f32 denormals inside those compares in every mode); the shared f64 path (#573) ordered the bits but had none of the signalling-NaN rules. Both now use AtomicFloatBits; the translation marks the shared float atomics with the mode's flush in MemoryInfo::flushDenormals. The 29 integer 64-bit forms, EXEC masking, immediate offsets, out-of-range addresses (0, no write) and contention (256 lanes, serial order) were measured as well and match the recompiler already; ds_add_f32 is not measured and is unchanged.

TechnicalDebt: the 64-bit entry loses its "not measured" sentence and gains the measured facts about mixed-width serialisation (hardware serialises 32-bit and 64-bit atomics on one address, the lock does not: in the measurement RADV lost 64 of 128 increments in wave32 and 32 in wave64, NVIDIA 48 in wave32, timing-dependent) and the misaligned fault; a new LDS float atomics entry records the rule; the f32 denormals entry points to it.

Measurement data, scripts and the host comparison before and after the change: https://github.com/SleglJan/AnyPS5/tree/audit/oracle-2026-10-08/oracle-audit/lds64 (report.md there is the summary).

Overlap: #2162 adds a sentence to the same f32 denormals TechnicalDebt entry (its v_fma_mixlo_f16/v_fma_mixhi_f16 flush rule), and #2025 and #1290 add entries next to the edited ones; none of them shares code with this PR or is built on by it, so whichever merges second rebases the one line. No open pull request changes the shared float atomics.

Tested

Linux (CachyOS, kernel 7.2.9, GCC 16.2.1), Release, ANYPS5_ENABLE_SPIRV_TOOLS=ON, RTX 3080 (615.71.09) and RADV (Mesa 26.2.4), on this branch (based on main d70b899).

• New agc_driver_lds_float_atomic_modes: 32 operand pairs per type (the classes above, both orders), each lane runs the six f32 and six f64 forms on its LDS slot; 27 result dwords per lane pinned bit for bit against the oracle in the titles' mode (0xc0, also used for "no float mode"), with both fields kept (0xf0), both flushed (0x00), both at 1 (0x50, kept) and both at 2 (0xa0, flushed). Written first: it failed on main's lowering at lane 7 (ds_min_rtn_f32, flushed denormal) and passes with the change.
• agc_driver_lds_atomics64: its CPU model gets the signalling-NaN rule, and the lanes of kind 2 now take their data from floats[(tid / 4 + 9) % 12], which feeds the table's signalling NaN, −inf and the negative denormal into min/max (the old index never reached entry 9). It failed on main at lane 2 (ds_min_f64, the signalling-NaN data not stored quieted) and passes with the change.
• Mutation checks, each fails: the 64-bit path without the signalling-NaN rules (both tests), the flush disabled (new test, lane 7), the old signalling NaN preferred over the data (new test, lane 22).
• The measured matrices replayed through the recompiler on both GPUs with the change: 0 of 35,072 result dwords differ from the oracle over the 256 lanes (f64 min/max 3,072 × 4 modes, f64 cmpst 1,536 × 4, f32 min/max/cmpst 2,304 × 4, 29 integer forms 7,424); before the change, per mode, 472 (f64 min/max, kept), 584 (f64 min/max, flushed), 40 (f64 cmpst, flushed), 336 (f32, kept) and 412 (f32, flushed) dwords differed, the same on both GPUs.
• agc_driver_lds_atomics, agc_driver_lds_integer_atomics, agc_driver_lds_condxchg, agc_driver_lds_read_write, agc_driver_lds_src2 pass on both GPUs; LDS tests --repeat until-fail:3 on both; full ctest 516/517 on each GPU; the one failure differs per GPU and runs no LDS instruction: agc_driver_storage_cache_budget on the iGPU (its memory budget) and agc_driver_dcc_keys_first_write on the RTX 3080 ("an unchanged pending image was uploaded again", also reported in several open pull requests today).
• tools/check_conventions.py --base upstream/main, tools/check_test_registration.py pass.
• Head 59e1899 (review follow-up: flush for field 2 as well, the 0x50 and 0xa0 modes): agc_driver_lds_float_atomic_modes and agc_driver_lds_atomics64 pass three repeats on RADV and the RTX 3080; the new 0xa0 mode fails without the change (lane 7, ds_max_rtn_f32); conventions and test registration pass. Fork Build on head 59e1899 merged into main a291ae0, run 38030543364: relinker green on all three platforms, linux (lavapipe) 539 passed, 29 skipped, 1 failed of 569 with both LDS tests passing (the failure is guest_video_out_resolution_status aborts on the linux CI job since #1714 merged: sceVideoOutOpen creates the window (#1367) and the test has no display guard #2509 on that base, fixed on main since); windows stops at Windows build fails on main: duplicate patched export name: LibwuIonIBw in libSceVideoOut (since #1339) #2462's build break on that base.
• Windows: the fork's Build workflow on head 3e8d755 merged onto main d70b899, run 37947911451: Windows builds and passes 508/508 with the AGC execution tests skipped (no Vulkan device on the runner, as for every execution test); the Linux job runs the new test on lavapipe (3.3 s) and passes 517/517; the relinker matrix passes on all three systems..
• Native Windows (popcoder565, RTX 3060 Laptop, NVIDIA 616.92, head 3e8d755 on main f8c2f07): agc_driver_lds_atomics64 and agc_driver_lds_float_atomic_modes pass; full suite 512 of 513, the one failure an intermittent test from fix(agc): storage access to a depth surface's stencil plane through its stencil aspect #2325.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none. The lock's mixed-width gap measured alongside is 64-bit LDS atomics under the workgroup lock lose concurrent 32-bit atomics on the same dwords, which the hardware serialises #2297.
• AI-assisted: yes (Claude Code)

🤖 Generated with Claude Code